The No Free Lunch Theorems for Optimisation: An Overview

Many algorithms have been devised for tackling combinatorial optimisation problems (COPs). Traditional Operations Research (OR) techniques such as Branch and Bound and Cutting Planes Algorithms can, given enough time, guarantee an optimal solution as they explicitly exploit features of the optimisation function they are solving. Specialised heuristics exist for most COPs that also exploit features of the optimisation function to arrive at a good, but probably not optimal, solution. There also exist a number of metaheuristics, algorithms that describe how to search the solution space without being tied to any one problem type. Some of these may be tailored to make them suit a particular problem better. However, there are a number of so-called “black-box” optimisation algorithms, which use little or no problem-specific information. Key examples of the black-box approach are Evolutionary Algorithms (EAs) and Simulated Annealing (SA). Random search is also a black-box optimisation algorithm, and so represents an important benchmark against which the performance of other search algorithms may be measured. Given an optimisation problem (i.e. objective function) f and an algorithm a, it is important to have some measure of how well a performs on f . Moreover, given empirical evidence of a’s performance on f , is it possible to make generalisations about a’s performance on other functions, either of the same or different type as f? Intuition would have one believe that there are some algorithms that will perform better than others on average. However, the No Free Lunch (NFL) Theorems state that such an assertion cannot be made. That is, across all optimisation functions, the average performance of all algorithms is the same. This means that if an algorithm performs well on one set of problems then it will perform poorly (worse than random search) on all others. To put the NFL theorems in their proper context, it is important to understand what they are and are not saying. From [2], p.69: